Renumbering unstructured grids to improve the performance of codes on hierarchical memory machines

The performance of unstructured grid codes on workstations and distributed memory parallel computers is substantially affected by the efficiency of the memory hierarchy. This efficiency essentially depends on the order of computation and numbering of the grid. Most grid generators do not take into account the effect of the memory hierarchy when producing grids so application programmers must renumber grids to improve the performance of their codes. To design a good renumbering scheme a detailed runtime analysis of the data movement in an application code is needed. Thus, a memory hierarchy simulator has been developed to analyse the effect of existing renumbering schemes such as bandwidth reduction, the Greedy method, colouring, random numbering and the original numbering produced by the grid generator. The renumbering is applied to either vertices, edges, faces or cells and two algorithms are proposed to consistently renumber the other entities used in the solver. The simulated and actual timings show that bandwidth reduction and Greedy methods give the best performance on IBM RS/6000, SGI Indy, SGI Indigo and SGI Power Challenge machines for three-dimensional Poissons’s, Maxwell’s and the Euler equations solvers. The improvement in performance is over a factor of two for applications with large grids and a high ratio of memory-accesses to computation. This factor is even higher for memory hierarchies with small caches.